Split Federated Learning
Split federated learning is a cross-field of FL and split learning, which can be regarded as a subset of distributed learning with some special settings.
Table of Contents
1. Split Federated Learning
1.1. Algorithm
1.1.1. SFLv1 Workflow
This table illustrates the procedure of computation and communication for each epoch.
- client
- main server (stores part of model; do forward and backward propagation)
- fed server (solely for aggregating client-side models)
| Clients | Server |
|---|---|
| Forwardprop @ local (in parallel) | |
| Send activition to server | |
| Do rest of forwardprop (for each client) | |
| Do part of backprop (for each client) | |
| Send gradient to respective clients | |
| Update server model (e.g. agg. with FedAvg) | |
| Do backprop, computes gradient | |
| Send gradient back to server with DP | |
| Aggregate client gradients (e.g. with FedAvg) | |
| Send aggregated gradient to clients | |
| Update param with received gradient |
After several epoches, which is called a round, all client-side models are sent to fed-server for aggregation (e.g., with FedAvg).
SFLv1 is server-side aggregation based variant. It computes forward and backward propagation for each client in parallel and aggregates gradients with FedAvg to update server-side model *once.
1.1.2. SFLv2 Workflow
In contrast, SFLv2 does not depend on aggregation. After receiving client-side activition, it sequentially processes the forward and backward propagation.
1.1.3. Label Sharing
In machine learning, due to privacy concerns, algorithms are also developed to take cases that disabling label-sharing into account.
If label-sharing is allowed, then we can simply adopt SFLv1 or SFlv2. In cases that label-sharing is banned, we can make client process 2 model portions:
- part one: first few layers of the model \(W\)
- part two: last few layers of the model \(W\) and loss calculation.
This settings trade communication expense for better privacy concerns.